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The invention disclosed in this document resulted from 
research in aeronautical and space activities performed under 
programs of the National Aeronautics and Space Administration. 
The invention is owned by NASA and is, therefore, available for 
licensing in accordance with the NASA Patent Licensing 
Regulation (14 Code of Federal Regulations 1245.2). 

To encourage commercial utilization of NASA-owned inventions, 
it is NASA policy to grant licenses to commercial concerns. 
Although NASA encourages nonexclusive licensing to promote 
competition and achieve the widest possible utilization, NASA 
will consider the granting of a limited exclusive license, 
pursuant to the NASA Patent Licensing Regulations, when such a 
license will provide the necessary incentive to the licensee to 
achieve early practical application of the invention. 


Address inquiries and all applications for license for this 
invention to NASA Resident Office-JPL, NASA Patent Counsel, 
Mail Code 180-801, 4800 Oak Grove Dr., Pasadena, CA 91103. 
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exclusive license are available from the above address. 
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Descript 

REAL-TIME OPTICAL MULTIPLE OBJECT 
RECOGNITION AND TRACKING SYSTEM AND METHOD 

Origin of the Invention ; 

The invention described herein was made in the 
performance of work under a NASA contract and is subject to 
the provisions of Public Law 96-517 (35 USC 202) in which 

the Contractor has elected not to retain title. 

Technical Field : 

The present invention relates to tracking systems and, 
more particularly, to an optical tracking system for 
recognizing and tracking multiple objects in real time. 

Description of the Prior Art : 

Object detection and tracking as accomplished in 
military command and control systems and air traffic control 
systems is a straightforward problem that has been 
accomplished with the combination of radar (for detecting) 
and digital computers (for tracking). All that is required 
in such applications is the detection of one or more objects 
within an area of two-dimensional or three-dimensional space 
from the radar source. 

The combination of object recognition and tracking, 
however, is another matter. A prior art system for 

accomplishing such as combination of functions is shown in 
simplified block diagram form in Figure 1. Assuming that a 
triangular shaped object 10 and a circular shaped object 12 
are moving within a two-dimensional area 14 they can be 
recognized and tracked within the area 14 by the components 
shown in Figure 1 according to prior art techniques. The 
area 14 is divided into an MxN matrix of discreetly 
addressable locations or pixels and viewed by a television 
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REAL-TIME OPTICAL MULTIPLE OBJECT 
RECOGNITION AND TRACKING SYSTEM AND METHOD 


The present invention has provided an optical recognition and 
tracking capability by apparatus and associated methods for 
recognizing a plurality of objects within a field of vision. The 
ability to track the recognized objects is also provided. Multiple 
point spatial filters are employed pre-defining the objects to be 
recognized at run-time. 

There is a laser (46) for producing a coherent laser beam (48). 
A beam splitter (24) is disposed to split the laser beam (48) into an 
object beam (26) and a reference beam (28). Beam expanders (50) and 
collimators (52) are positioned to transform the beams (26, 28 ) into 
coherent collimated light beams (26* , 28'). A two-dimensional spatial 
light modulator (54) is disposed in the object beam (26') to modulate 
the object beam (26‘) with optical information as a function of 
signals connected to X and Y inputs thereof. There is a first 
television camera (16) for developing X and Y signals reflecting the 
contents of its field of vision. The X and Y signals are connected to 
the X and Y inputs of the two-dimensional spatial light modulator 
(54). A hololens (38) is positioned in the object beam (26') 
subsequent to the two-dimensional spatial light modulator (54) to 
focus the object beam (26') at a plurality of focal points (42). A 
planar transparency-forming film 32 is disposed with the focal points 
(42) on an exposable surface thereof to form a multiple position 
interference filter (62) upon exposure of the exposable surface and 
subsequent development processing of the film 32. A reflector (53) is 
positioned to direct the reference beam (28') onto the exposable 
surface of the film 32 to expose the exposable surface in combination 
with images focused by the hololens (38) to form interference patterns 
on the exposable surface. Finally, there is apparatus (16', 64) for 
sensing and indicating light passage through respective ones of the 
positions of the multiple position interference filter (62) following 
development processing of the exposable surface whereby recognition of 
objects corresponding to respective ones of the positions of the 
multiple position interference filter (62) is affected. For tracking, 
there is apparatus (64) for focusing light passing through the 
interference filter (62) onto a matrix of charge coupled devices in a 
second television camera (16') to form a two-dimensional display of 
the recognized objects within the field of view of the first 
television camera (16). 

This invention takes the basic technology of a Vander Lugt filter 
and adds a hololens (co- invented by H. K. Liu) to provide a system 
which can recognize multiple objects optically in real-time — which 
the prior art has been unable to do. The technique replaces time-, 
space-, and cost-intensive digital techniques. In place of multiple 
objects, the system can also recognize multiple orientations of a 
single object - see Figure 7. This latter capability has potential 
for space applications where space and weight are at a premium. 
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camera 16. The information from the television camera 16 
relative to the pixel positions within the area 14 is output 
to a computer 18 on a periodic basis. The period of the 
output, of course, is established as a function of the speed 
of movement of the objects 10, 12. For slow-moving objects, 
the period or sampling rate is long while, on the other 
hand, for fast-moving objects the sampling rate or period 
between data output must be short. The computer 18 stores 
the data from the camera 16 in a memory matrix 20 containing 
a historical file of the data for the present time (n), the 
previous time (n-1), etc. The size or number of the 
historical snapshots is dependent upon the amount of 
information relative to the object movement that is desired. 
If only instantaneous relative movement since the last data 
is required, only a single history of time n-1 is required. 
If a more detailed tracking profile is required, obviously 
more data must be retained. In any event, those skilled in 
the art will appreciate that in order to "recognize" objects 
10, 12 employing the digital computer approach of Figure 1 a 
detailed programming task must be undertaken relative to the 
data in each instance. First, the edges of the objects 10, 
12, must be located and defined. Having thus established 
the edge locations, they must then be analyzed to determine 
whether they are objects to be recognized, i.e. a triangle 
or a circle. Such a digital approach to object recognition 
and tracking is computationally intensive, requiring large 
quantities of time, space, and money to accomplish it. 
Obviously, objects more complex in shape than a simple 
triangle or circle intensify the problem. 

On the other side of the coin, object recognition using 
optical techniques is known in the art. As shown in Figure 
2, a collimated beam of light 22 impressed with a known 
pattern (e.g. the letter "A") can be split by a beam 
splitter 24 into an object beam 26 and a reference beam 28. 
If the object beam 26 is then focused by lens 30 onto a 
transparent film 32 and, simultaneously, the reference beam 
28 is used to illuminate the film 32 after traveling a 
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substantially identical path length to that of the object 
beam 26, the film 32 will be exposed with an interference 
pattern of the letter A. If, subsequently, a light beam 22' 
as shown in Figure 2B is focused onto the developed film 32' 
(acting as a spatial filter) and the output, if any, is 
refocused by a second lens 34, a spot 36 will appear if, and 
only if, there is correlation between the image impressed on 
the light beam 22' (in this example another letter A) and 
the previously established interference pattern on the film 
32'. This is known as a Vander Lugt filter, which is well 
known in the art. With the advent of dynamically modifiable 
spatial light filter elements such as liquid crystal and 
magneto-optic filters, the film 32 has been replaced by such 
devices in tested systems so that object recognition can 
take place in real-time by driving the dynamic spatial 
filter with a series of pre-established interference 
patterns. Unfortunately, there is nothing in the prior art 
which provides the capability of employing such optical 
object recognition techniques with the ability to track the 
objects once recognized. 

Disclosure of the Invention : 

The present invention has provided an optical 
recognition and tracking capability by the apparatus for 
recognizing a plurality of objects within a field of vision 
comprising, a laser producing a coherent laser beam; a beam 
splitter disposed to split the laser beam into an object 
beam and a reference beam; a pair of beam expanders and 
collimators positioned to transform the object and reference 
beams into coherent collimated light beams; a two- 
dimensional spatial light modulator disposed in the object 
beam to modulate the object beam with optical information as 
a function of signals connected to X and Y inputs thereof; a 
television camera for developing X and Y signals reflecting 
the contents of its field of vision, the X and Y signals 
being connected to the X and Y inputs of the two-dimensional 
spatial light modulator; a hololens positioned in the object 
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beam subsequent to the two-dimensional spatial light 
modulator to focus the object beam at a plurality of focal 
points; a planar transparency-forming film disposed with the 
focal points on an exposable surface thereof to form a 
multiple position interference filter upon exposure of the 
exposable surface and subsequent development processing of 
the film; means for directing the reference beam onto the 
exposable surface of the film to expose the exposable 
surface in combination with images focused by the hololens 
to form interference patterns on the exposable surface; and, 
means for sensing and indicating light passage through 
respective ones of the positions of the multiple position 
interference filter following development processing of the 
exposable surface whereby recognition of objects 
corresponding to respective ones of the positions of the 
multiple position interference filter is affected. 

In the preferred embodiment, the means for sensing and 
indicating light passage through respective ones of the 
positions of the multiple position interference filter 
includes a second television camera containing a matrix of 
charge coupled devices and means for focusing light passing 
through the interference filter onto the matrix. 

Additionally in the preferred embodiment, the means for 
sensing and indicating light passage through respective ones 
of the positions of the multiple position interference 
filter employs outputs of the charge coupled devices to 
indicate light passage at each position of the interference 
filter and includes means for indicating light passage only 
if light passing through a position is above a pre- 
established threshold level corresponding to object 
recognition . 

For tracking of one or more objects, light passing 
through positions above the pre-established threshold level 
is focused onto a second television camera to create a two- 
dimensional image of the field of view of the first 
television camera and the objects to be tracked therein. 
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Detailed Description of the Drawings ; 

Figure 1 is a simplified drawing showing a prior art 
approach to using a digital computer and television camera 
to recognize and track objects. 

Figure 2A is a simplified drawing showing the prior art 
technique of creating a Vander Lugt filter to be used in 
object recognition. 

Figure 2B shows the way that the Vander Lugt filter of 
Figure 2A is employed to recognize an object optically. 

Figure 3 is a simplified drawing showing the manner in 
which a multiple focus hololens produces multiple focal 
points of the same image. 

Figure 4 is a simplified perspective drawing showing 
the apparatus employed in the present invention during 
filter preparation and pre-definition. 

Figures 5(A)-5(D) comprise a sequence showing how the 
apparatus of Figure 4 is employed to create a multiple 
object spatial filter array according to the present 
invention . 

Figure 6 shows the manner in which the apparatus of 
Figure 4 is configured during real-time recognition and 
tracking of multiple objects optically employing the spatial 
filter array constructed according to the method of Figures 
5 (B ) -5 (D ) . 

Figure 7 is a drawing showing how the present invention 
can be employed to recognize the orientation of a spacecraft 
as the multiple objects of a pre-established filter. 

Detailed Description of the Invention ; 

One of the most important potential applications of 
optical processing in addition to pattern recognition is 
object tracking. In recent years, the rapid development of 
various spatial light modulators such as the liquid crystal 
light cell, the magneto-optic spatial light modulator, the 
micro-channel spatial light modulator, and the liquid 
crystal television spatial light modulator (LCTVSLM) has 
expedited progress in the development of system 
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architectures for real-time optical pattern recognition and 
tracking. To date, however, only a real-time optical 

tracking system capable of tracking a single object has been 
demonstrated in the prior art. 

5 In 1982 and 1983, co-inventor H. K. Liu of this 

application (and others) reported the development and 
fabrication of a multi-focus hololens using dichromated 
gelatin. A hololens, generally indicated as 38, is shown in 
simplified form in Figure 3. It is characteristic of the 
10 hololens 38 that when a collimated beam of light 40 passes. 

therethrough, a plurality of focal points 42 are created. 
It is important to recognize that each focal point 42 

contains all the information from the collimated light beam 
40. Thus, if the light beam 40 had a triangle such as 
15 object 10 impressed thereon (as by passing through a 

transparent filter containing such an object image) each 
focal point 42 would contain the image of the entire 
triangle. It is the incorporation of the hololens into the 
basic Vander Lugt filter apparatus of Figures 2 a and 2B 
20 which makes the present invention possible. 

The basic system of the present invention as employed 
for pre-establishing or pre-defining the objects to be 

recognized is shown in Figure 4 wherein it is generally 

indicated as 44. A laser 46 is employed to produce a beam 
25 of coherent light 48. The beam 48 is passed through a beam 

splitter 24 to produce an object beam 26 and a reference 
beam 28. Each beam 26, 28 is passed through a beam expander 
50 and a collimating lens 52 to produce expanded and 
collimated beams 26' and 28'. The reference beam 28' is 

30 reflected at 53 onto a transparent film 32 in the same 

manner as with the Vander Lugt filter of Figures 2A and 2B 
and for the same purpose. Note that for simplicity and 
convenience only, such terms as "film" and "development" are 
employed in this description and the claims which follow. 
35 As those skilled in the art will recognize, the creation of 

the transparent interference filters employed in the prior 
art Vander Lugt filter and this invention can be created by 
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various technical processes which form no part of the points 
of novelty of the present invention. Simultaneously, a 
television camera 16 (preferably of the charge coupled 
device (CCD) variety for simplicity and smallness of size) 
is used to drive an LCTVSLM 54 disposed in the path of the 
object beam 26'. Thus, whatever is viewed by the camera 16 
is impressed onto the object beam 26' by the LCTVSLM 54. 
Immediately following the LCTVSLM 54, the object beam 26' is 
low-pass filtered at 56 to remove the periodical spectrum 
pattern associated with the grid structure of the LCTVSLM 
54. The filtered object beam 26' is then passed through a 
multiple-focus hololens 38 to create an NxN array of focal 
points 42 on the . film 32. As described above with respect 
to the Vander Lugt filter, the interference pattern between 
the reference beam 28' and the multiple-focal points 42 
creates an NxN array Fourier spectra of the input scene. 
Without more, the multi-point spatial filter created by 
development of the film 32 would be of no more use than the 
single point filter produced by the prior art. As shown in 
Figures 5(A)-5(D), however, during synthesis of multiple, 
matched, spatial filters (MSFs) on. the film 32, the 
reference beam 28' is linearly shifted to spatially separate 
the output correlation planes associated with each MSF such 
that recognition and tracking of multiple objects can 
subsequently take place in real-time. By way of example, 
assume that the hololens 38 creates nine focal points 42 in 
a 3x3 matrix as shown in Figure 5A. Assume for simplicity 
that three objects are to be recognized and tracked, the 
triangle 10, the circle 12, and a rectangle 58. Using the 
apparatus of Figure 4, the triangle 10 is first placed 
within the viewing area of the television camera 16 while a 
shield 60 is placed over the unexposed film 32 covering all 
but one of the focal points 42. The laser 46 is then 
operated to expose the focal point 42 to be associated with 
the triangle 10 as shown in Figure 5B. The process is then 
repeated as shown in Figure 5C employing the rectangle 58 
and a shield 60' exposing only another of the focal points 
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42 to be associated with the rectangle 58. Finally, as 
shown in Figure 5D, the process is repeated once again with 
a third shield 60 ' ' exposing a third focal point 42 on the 
film 32 to be associated with the circle 12. For additional 
objects to be recognized, the process would be repeated in 
like fashion. Again, as those skilled in the art will 

appreciate, in a commercial embodiment of the present 
invention the shields 60, 60', 60' ' of the laboratory 
testing embodiment could be replaced by a liquid crystal 
matrix over the film used to block all the focal points but 
the particular one of interest at each instance employing 
dynamic optical techniques. For simplicity, a 3x3 or 9- 
focus hololens is employed in the example. A 25-focus 
hololens system has been demonstrated and the state of the 
art techniques available at present can increase the number 
up to 100. It is expected that in the future this number 
will increase dramatically. 

Having thus created the array of prefabricated 
holographic matched spatial filters, real-time optical 
multiple object recognition and tracking is accomplished by 
the system 44' as shown in Figure 6 wherein the 
prefabricated filter is indicated as 62. As the multiple 
objects 10, 12, 58 move within the field of vision of the 
television camera 16, their images are impressed on the 
object beam 26* by the LCTVSLM 54. The scene of the 
multiple objects is duplicated by the multiple focus 
hololens 38 at the nine focal points 42 with the image of 
"recognized" objects passing through the filter 62 at each 
object's associated focal point location. The output from 
the filter 62 is focused by lens 64 into a second television 
camera 16 ' The output correlation peaks are picked up by 
the array of CCD detectors within the camera 16' for 
threshold detection. The level of light passage through 
each focal point 42 of the filter 62 can be analyzed and 
determined from the CCD outputs from the camera 16' by 
techniques well known to those skilled in the art. 
Obviously, objects within the field of view of the camera 16 
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(such as triangle 10, circle 12, and rectangle 58) having 
matched spatial filters within the filter 62 will pass their 
image through to the lens 64 and thence to the television 
camera 16'. Other objects within the field of view, having 
no matched filter, will produce no image. Thus, detection 
of an image passing through a given focal point 42 on the 
filter 62 above a threshold level (some light will pass 
through all points of the filter 62 even in a non-recognized 
state) indicates the associated object is present in the 
field of view. The image from "recognized" focal points 
can, of course, be used to track the associated object 
within the field of view as it is only of the recognized 
object. The combined image of multiple recognized objects 
can, of course, be formed, if desired, by re-focusing the 
multiple images at a single focal point. 

In the laboratory experimental setup of the inventors 
herein, a Radio Shack® liquid crystal television was 
disassembled and the screen thereof used for the LCTVSLM 54. 
Phase non-uniformity was eliminated by submerging the 
LCTVSLM into a liquid gate filled with insulating mineral 
oil. The critically important shift invariance was obtained 
satisfactorily in this manner. A 3x3 focus dichromated 
gelatin hololens was used and, for simplicity of 
demonstration, a column of three matched spatial filters 
were synthesized to track the motions of three independent 
objects. The matched spatial filters were recorded using a 
thermal plastic plate for high diffraction efficiency and 
ease in processing. In the experimental demonstrations, 
three different toy cars were selected as the input objects. 
The cars were photographed from above heading in 45° 
directions. The range of the television camera coverage was 
about 75°, however, the range of tracking of the input 
object was limited to about 30'. This is mainly because of 
the limited aperture size of the hololens (0.3 cm) as 
compared to the television screen size (5.4 cm by 4.4 cm). 
Employing the above-described apparatus, the inventors 
demonstrated the first known optical multiple object 
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tracking system. The system is capable of simultaneous 
tracking of multiple objects, each with independent 
movements in real-time, limited only to the television frame 
rate (30 msec). In order to perform a similar tracking 
operation employing a digital computer, a large counter 
system and very complex software would be needed. Although 
the inventors have demonstrated the tracking of only three 
objects, the system capacity can easily be expanded by two 
orders of magnitude. 

It should be recognized by those skilled in the art 
that in recognizing multiple "objects", the system and 
method of the present invention also provide the ability to 
recognize different orientations of a single object as 
depicted in Figure 7. Such an ability could be used to good 
advantage, for example, in space applications such as the 
control of docking maneuvers, and the like. Thus, for 
example, the filter 62 can comprise multiple interference 
patterns at the focal points 42 corresponding to different 
orientations of the spacecraft 66. Thus, as passage of the 
beam 26' occurs through the focal point 42 labelled "B" and 
is sensed by the CCDs within the camera 16' in a manner well 
known to those skilled in the art, the system of the present 
invention recognizes that the spacecraft 66 is oriented with 
its nose directly at the camera 16. Employing the image 
passing through the "B" focal point of the filter 62, of 
course, the spacecraft 66 can be tracked within the field of 
vision of the camera 16 since the entire field of vision is 
contained within the beam 26' passing through that focal 
point 42 because of the characteristics of the hololens 38 
as described previously herein. If the spacecraft 66 turns 
180° and heads away from the camera 16, this, of course, 
will be recognized by the focal point 42 labelled "E" 
passing the beam 26' therethrough. All this could be 
performed optically with a minimum of equipment as compared 
to digital techniques, as is of prime importance in space 
applications. 
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REAL-TIME OPTICAL MULTIPLE OBJECT 
RECOGNITION AND TRACKING SYSTEM AND METHOD 

Apparatus and associated methods for optically 
recognizing a plurality of objects within a field of vision 
and tracking them if desired. There is a laser (46) for 
producing a coherent laser beam (48). A beam splitter (24) 
is disposed to split the laser beam (48) into an object beam 
(26) and a reference beam (28). Beam expanders (50) and 
collimators (52) are positioned to transform the beams (26, 
28) into coherent collimated light beams (26', 28'). A two- 
dimensional spatial light modulator (54) is disposed in the 
object beam (26') to modulate the object beam (26') with 
optical information as a function of signals connected to X 
and Y inputs thereof. There is 'a first television camera 
(16) for developing X and Y signals reflecting the contents 
of its field of vision. The X and Y signals are connected 
to the X and Y inputs of the two-dimensional spatial light 
modulator (54). A hololens (38) is positioned in the object 
beam (26') subsequent to the two-dimensional spatial light 
modulator (54) to focus the object beam (26‘) at a plurality 
of focal points (42). A planar transparency-forming film 32 
is disposed with the focal points (42) on an exposable 
surface thereof to form a multiple position interference 
filter (62) upon exposure of the exposable surface and 
subsequent development processing of the film 32. A 
reflector (53) is positioned to direct the reference beam 
(28') onto the exposable surface of the film 32 to expose 
the exposable surface in combination with images focused by 
the hololens (38) to form interference patterns on the 
exposable surface. Finally, there is apparatus (16', 64) 
for sensing and indicating light passage through respective 
ones of the positions of the multiple position interference 
filter (62) following development processing of the 
exposable surface whereby recognition of objects 
corresponding to respective ones of the positions of the 
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multiple position interference filter (62) is affected. For 
tracking, there is apparatus (64) for focusing light passing 
through the interference filter (62) onto a matrix of charge 
coupled devices in a second television camera (16') to form 
a two-dimensional display of the recognized objects within 
the field of view of the first television camera (16). 



